Architecture for Responding to Visual Query

ABSTRACT

A visual query such as a photograph, a screen shot, a scanned image, a video frame, or an image created by a content authoring application is submitted to a visual query search system. The search system processes the visual query by sending it to a plurality of parallel search systems, each implementing a distinct visual query search process. These parallel search systems may include but are not limited to optical character recognition (OCR), facial recognition, product recognition, bar code recognition, object-or-object-category recognition, named entity recognition, and color recognition. Then at least one search result is sent to the client system. In some embodiments, when the visual query is an image containing a text element and a non-text element, at least one search result includes an optical character recognition result for the text element and at least one image-match result for the non-text element.

RELATED APPLICATIONS

This application claims priority to the following U.S. ProvisionalPatent Applications all of which are incorporated by reference herein intheir entirety: U.S. Provisional Patent Application No. 61/266,116,filed Dec. 2, 2009, entitled “Architecture for Responding to a VisualQuery,” and U.S. Provisional Patent Application No. 61/232,397, filedAug. 7, 2009, entitled “Architecture for Responding to a Visual Query.”

This application is related to the following U.S. Provisional PatentApplications all of which are incorporated by reference herein in theirentirety: U.S. Provisional Patent Application No. 61/266,122, filed Dec.2, 2009, entitled “User Interface for Presenting Search Results forMultiple Regions of a Visual Query;” U.S. Provisional Patent ApplicationNo. 61/266,125, filed Dec. 2, 2009, entitled “Identifying MatchingCanonical Documents In Response To A Visual Query;” U.S. ProvisionalPatent Application No. 61/266,126, filed Dec. 2, 2009, entitled “Regionof Interest Selector for Visual Queries;” U.S. Provisional PatentApplication No. 61/266,130, filed Dec. 2, 2009, entitled “ActionableSearch Results for Visual Queries;” U.S. Provisional Patent ApplicationNo. 61/266,133, filed Dec. 2, 2009, entitled “Actionable Search Resultsfor Street View Visual Queries,” and U.S. Provisional Patent ApplicationNo. 61/266,499, filed Dec. 3, 2009, entitled “Hybrid Use Location SensorData and Visual Query to Return Local Listing for Visual Query.”

TECHNICAL FIELD

The disclosed embodiments relate generally to a server systemarchitecture encompassing a plurality of parallel search systems forprocessing a visual query.

BACKGROUND

A text-based or term-based searching, wherein a user inputs a word orphrase into a search engine and receives a variety of results is auseful tool for searching. However, term based queries require that auser be able to input a relevant term. Sometimes a user may wish to knowinformation about an image. For example, a user might want to know thename of a person in a photograph, or a user might want to know the nameof a flower or bird in a picture. Accordingly, a system that can receivea visual query and provide search results would be desirable.

SUMMARY

According to some embodiments, there is computer-implemented method ofprocessing a visual query at a server system. A visual query is receivedfrom a client system. The visual query is processed by sending thevisual query to a plurality of parallel search systems for simultaneousprocessing. Each of the plurality of search systems implements adistinct visual query search process of a plurality of visual querysearch processes. The plurality of visual query search processesincludes at least: optical character recognition (OCR), facialrecognition, and a first query-by-image process other than OCR andfacial recognition. A plurality of search results is received from oneor more of the plurality of parallel search systems. At least one of theplurality of search results is sent to the client system.

In some embodiments, the method further includes, when at least two ofthe received search results meet predefined criteria, ranking thereceived search results that meet the predefined criteria, and sendingat least one search result of the ranked search results to the clientsystem.

In some embodiments, the first query-by-image process is productrecognition, bar code recognition, object-or-object-categoryrecognition, named entity recognition, or color recognition.

In some embodiments, the visual query is a photograph, a screen shot, ascanned image, or a video frame. The client system can be a mobiledevice, a desktop device, or other device.

In some embodiments, the visual query is received from a clientapplication executed by the client system, such a search application, asearch engine plug-in for a browser application, or a search engineextension for a browser application. In some embodiments, the visualquery is received from a content authoring application executed by theclient system.

When the visual query is an image containing a text element and anon-text element, in some embodiments, the search result includes anoptical character recognition result for the text element and at leastone image-match result for the non-text element.

In some embodiments, when the visual query is an image containing a textelement and a non-text element, the search result includes aninteractive results document comprising a first visual identifier forthe text element with a link to a search result produced by an opticalcharacter recognition process, and a second visual identifier for thenon-text element with a link to a search result produced by animage-match process.

In some embodiments, the method further includes combining at least twoof the plurality of search results into a compound search result.

According to some embodiments, a search engine system is provided forprocessing a visual query. The system includes one or more centralprocessing units for executing programs and memory storing one or moreprograms to be executed by the one or more central processing units. Theone or more programs include instructions for performing the following.A visual query is received from a client system. The visual query isprocessed by sending the visual query to a plurality of parallel searchsystems for simultaneous processing. Each of the plurality of searchsystems implements a distinct visual query search process of a pluralityof visual query search processes. The plurality of visual query searchprocesses includes at least: optical character recognition (OCR), facialrecognition, and a first query-by-image process other than OCR andfacial recognition. A plurality of search results is received from oneor more of the plurality of parallel search systems. At least one of theplurality of search results is sent to the client system. Such a systemmay also include program instructions to execute the additional optionsdiscussed above.

According to some embodiments, a computer readable storage medium systemfor processing a visual query is provided. The computer readable storagemedium stores one or more programs configured for execution by acomputer, the one or more programs comprising instructions forperforming the following. A visual query is received from a clientsystem. The visual query is processed by sending the visual query to aplurality of parallel search systems for simultaneous processing. Eachof the plurality of search systems implements a distinct visual querysearch process of a plurality of visual query search processes. Theplurality of visual query search processes includes at least: opticalcharacter recognition (OCR), facial recognition, and a firstquery-by-image process other than OCR and facial recognition. Aplurality of search results is received from one or more of theplurality of parallel search systems. At least one of the plurality ofsearch results is sent to the client system. Such a computer readablestorage medium may also include program instructions to execute theadditional options discussed above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a computer network that includesa visual query server system.

FIG. 2 is a flow diagram illustrating the process for responding to avisual query, in accordance with some embodiments.

FIG. 3 is a flow diagram illustrating the process for responding to avisual query with an interactive results document, in accordance withsome embodiments.

FIG. 4 is a flow diagram illustrating the communications between aclient and a visual query server system, in accordance with someembodiments.

FIG. 5 is a block diagram illustrating a client system, in accordancewith some embodiments.

FIG. 6 is a block diagram illustrating a front end visual queryprocessing server system, in accordance with some embodiments.

FIG. 7 is a block diagram illustrating a generic one of the parallelsearch systems utilized to process a visual query, in accordance withsome embodiments.

FIG. 8 is a block diagram illustrating an OCR search system utilized toprocess a visual query, in accordance with some embodiments.

FIG. 9 is a block diagram illustrating a facial recognition searchsystem utilized to process a visual query, in accordance with someembodiments.

FIG. 10 is a block diagram illustrating an image to terms search systemutilized to process a visual query, in accordance with some embodiments.

FIG. 11 illustrates a client system with a screen shot of an exemplaryvisual query, in accordance with some embodiments.

FIGS. 12A and 12B each illustrate a client system with a screen shot ofan interactive results document with bounding boxes, in accordance withsome embodiments.

FIG. 13 illustrates a client system with a screen shot of an interactiveresults document that is coded by type, in accordance with someembodiments.

FIG. 14 illustrates a client system with a screen shot of an interactiveresults document with labels, in accordance with some embodiments.

FIG. 15 illustrates a screen shot of an interactive results document andvisual query displayed concurrently with a results list, in accordancewith some embodiments.

Like reference numerals refer to corresponding parts throughout thedrawings.

DESCRIPTION OF EMBODIMENTS

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the present invention. However, it will beapparent to one of ordinary skill in the art that the present inventionmay be practiced without these specific details. In other instances,well-known methods, procedures, components, circuits, and networks havenot been described in detail so as not to unnecessarily obscure aspectsof the embodiments.

It will also be understood that, although the terms first, second, etc.may be used herein to describe various elements, these elements shouldnot be limited by these terms. These terms are only used to distinguishone element from another. For example, a first contact could be termed asecond contact, and, similarly, a second contact could be termed a firstcontact, without departing from the scope of the present invention. Thefirst contact and the second contact are both contacts, but they are notthe same contact.

The terminology used in the description of the invention herein is forthe purpose of describing particular embodiments only and is notintended to be limiting of the invention. As used in the description ofthe invention and the appended claims, the singular forms “a,” “an,” and“the” are intended to include the plural forms as well, unless thecontext clearly indicates otherwise. It will also be understood that theterm “and/or” as used herein refers to and encompasses any and allpossible combinations of one or more of the associated listed items. Itwill be further understood that the terms “comprises” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in response to detecting,” dependingon the context. Similarly, the phrase “if it is determined” or “if isdetected” may be construed to mean “upon determining” or “in response todetermining” or “upon detecting (the stated condition or event)” or “inresponse to detecting (the stated condition or event),” depending on thecontext.

FIG. 1 is a block diagram illustrating a computer network that includesa visual query server system according to some embodiments. The computernetwork 100 includes one or more client systems 102 and a visual queryserver system 106. One or more communications networks 104 interconnectthese components. The communications network 104 may be any of a varietyof networks, including local area networks (LAN), wide area networks(WAN), wireless networks, wireline networks, the Internet, or acombination of such networks.

The client system 102 includes a client application 108, which isexecuted by the client system, for receiving a visual query (e.g.,visual query 1102 of FIG. 11). A visual query is an image that issubmitted as a query to a search engine or search system. Examples ofvisual queries, without limitations include photographs, scanneddocuments and images, and drawings. In some embodiments, the clientapplication 108 is selected from the set consisting of a searchapplication, a search engine plug-in for a browser application, and asearch engine extension for a browser application. In some embodiments,the client application 108 is an “omnivorous” search box, which allows auser to drag and drop any format of image into the search box to be usedas the visual query.

A client system 102 sends queries to and receives data from the visualquery server system 106. The client system 102 may be any computer orother device that is capable of communicating with the visual queryserver system 106. Examples include, without limitation, desktop andnotebook computers, mainframe computers, server computers, mobiledevices such as mobile phones and personal digital assistants, networkterminals, and set-top boxes.

The visual query server system 106 includes a front end visual queryprocessing server 110. The front end server 110 receives a visual queryfrom the client 102, and sends the visual query to a plurality ofparallel search systems 112 for simultaneous processing. The searchsystems 112 each implement a distinct visual query search process andaccess their corresponding databases 114 as necessary to process thevisual query by their distinct search process. For example, a facerecognition search system 112-A will access a facial image database114-A to look for facial matches to the image query. As will beexplained in more detail with regard to FIG. 9, if the visual querycontains a face, the facial recognition search system 112-A will returnone or more search results (e.g., names, matching faces, etc.) from thefacial image database 114-A. In another example, the optical characterrecognition (OCR) search system 112-B, converts any recognizable text inthe visual query into text for return as one or more search results. Inthe optical character recognition (OCR) search system 112-B, an OCRdatabase 114-B may be accessed to recognize particular fonts or textpatterns as explained in more detail with regard to FIG. 8.

Any number of parallel search systems 112 may be used. Some examplesinclude a facial recognition search system 112-A, an OCR search system112-B, an image-to-terms search system 112-C (which may recognize anobject or an object category), a product recognition search system(which may be configured to recognize 2-D images such as book covers andCDs and may also be configured to recognized 3-D images such asfurniture), bar code recognition search system (which recognizes 1D and2D style bar codes), a named entity recognition search system, landmarkrecognition (which may configured to recognize particular famouslandmarks like the Eiffel Tower and may also be configured to recognizea corpus of specific images such as billboards), place recognition aidedby geo-location information provided by a GPS receiver in the clientsystem 102 or mobile phone network, a color recognition search system,and a similar image search system (which searches for and identifiesimages similar to a visual query). Further search systems can be addedas additional parallel search systems, represented in FIG. 1 by system112-N. All of the search systems, except the OCR search system, arecollectively defined herein as search systems performing an image-matchprocess. All of the search systems including the OCR search system arecollectively referred to as query-by-image search systems. In someembodiments, the visual query server system 106 includes a facialrecognition search system 112-A, an OCR search system 112-B, and atleast one other query-by-image search system 112.

The parallel search systems 112 each individually process the visualsearch query and return their results to the front end server system110. In some embodiments, the front end server 100 may perform one ormore analyses on the search results such as one or more of: aggregatingthe results into a compound document, choosing a subset of results todisplay, and ranking the results as will be explained in more detailwith regard to FIG. 6. The front end server 110 communicates the searchresults to the client system 102.

The client system 102 presents the one or more search results to theuser. The results may be presented on a display, by an audio speaker, orany other means used to communicate information to a user. The user mayinteract with the search results in a variety of ways. In someembodiments, the user's selections, annotations, and other interactionswith the search results are transmitted to the visual query serversystem 106 and recorded along with the visual query in a query andannotation database 116. Information in the query and annotationdatabase can be used to improve visual query results. In someembodiments, the information from the query and annotation database 116is periodically pushed to the parallel search systems 112, whichincorporate any relevant portions of the information into theirrespective individual databases 114.

The computer network 100 optionally includes a term query server system118, for performing searches in response to term queries. A term queryis a query containing one or more terms, as opposed to a visual querywhich contains an image. The term query server system 118 may be used togenerate search results that supplement information produced by thevarious search engines in the visual query server system 106. Theresults returned from the term query server system 118 may include anyformat. The term query server system 118 may include textual documents,images, video, etc. While term query server system 118 is shown as aseparate system in FIG. 1, optionally the visual query server system 106may include a term query server system 118.

Additional information about the operation of the visual query serversystem 106 is provided below with respect to the flowcharts in FIGS.2-4.

FIG. 2 is a flow diagram illustrating a visual query server systemmethod for responding to a visual query, according to certainembodiments of the invention. Each of the operations shown in FIG. 2 maycorrespond to instructions stored in a computer memory or computerreadable storage medium.

The visual query server system receives a visual query from a clientsystem (202). The client system, for example, may be a desktop computingdevice, a mobile device, or another similar device (204) as explainedwith reference to FIG. 1. An example visual query on an example clientsystem is shown in FIG. 11.

The visual query is an image document of any suitable format. Forexample, the visual query can be a photograph, a screen shot, a scannedimage, or a frame or a sequence of multiple frames of a video (206). Insome embodiments, the visual query is a drawing produced by a contentauthoring program (736, FIG. 5). As such, in some embodiments, the user“draws” the visual query, while in other embodiments the user scans orphotographs the visual query. Some visual queries are created using animage generation application such as Acrobat, a photograph editingprogram, a drawing program, or an image editing program. For example, avisual query could come from a user taking a photograph of his friend onhis mobile phone and then submitting the photograph as the visual queryto the server system. The visual query could also come from a userscanning a page of a magazine, or taking a screen shot of a webpage on adesktop computer and then submitting the scan or screen shot as thevisual query to the server system. In some embodiments, the visual queryis submitted to the server system 106 through a search engine extensionof a browser application, through a plug-in for a browser application,or by a search application executed by the client system 102. Visualqueries may also be submitted by other application programs (executed bya client system) that support or generate images which can betransmitted to a remotely located server by the client system.

The visual query can be a combination of text and non-text elements(208). For example, a query could be a scan of a magazine pagecontaining images and text, such as a person standing next to a roadsign. A visual query can include an image of a person's face, whethertaken by a camera embedded in the client system or a document scanned byor otherwise received by the client system. A visual query can also be ascan of a document containing only text. The visual query can also be animage of numerous distinct subjects, such as several birds in a forest,a person and an object (e.g., car, park bench, etc.), a person and ananimal (e.g., pet, farm animal, butterfly, etc.). Visual queries mayhave two or more distinct elements. For example, a visual query couldinclude a barcode and an image of a product or product name on a productpackage. For example, the visual query could be a picture of a bookcover that includes the title of the book, cover art, and a bar code. Insome instances, one visual query will produce two or more distinctsearch results corresponding to different portions of the visual query,as discussed in more detail below.

The server system processes the visual query as follows. The front endserver system sends the visual query to a plurality of parallel searchsystems for simultaneous processing (210). Each search system implementsa distinct visual query search process, i.e., an individual searchsystem processes the visual query by its own processing scheme.

In some embodiments, one of the search systems to which the visual queryis sent for processing is an optical character recognition (OCR) searchsystem. In some embodiments, one of the search systems to which thevisual query is sent for processing is a facial recognition searchsystem. In some embodiments, the plurality of search systems runningdistinct visual query search processes includes at least: opticalcharacter recognition (OCR), facial recognition, and anotherquery-by-image process other than OCR and facial recognition (212). Theother query-by-image process is selected from a set of processes thatincludes but is not limited to product recognition, bar coderecognition, object-or-object-category recognition, named entityrecognition, and color recognition (212).

In some embodiments, named entity recognition occurs as a post processof the OCR search system, wherein the text result of the OCR is analyzedfor famous people, locations, objects and the like, and then the termsidentified as being named entities are searched in the term query serversystem (118, FIG. 1). In other embodiments, images of famous landmarks,logos, people, album covers, trademarks, etc. are recognized by animage-to-terms search system. In other embodiments, a distinct namedentity query-by-image process separate from the image-to-terms searchsystem is utilized. The object-or-object category recognition systemrecognizes generic result types like “car.” In some embodiments, thissystem also recognizes product brands, particular product models, andthe like, and provides more specific descriptions, like “Porsche.” Someof the search systems could be special user specific search systems. Forexample, particular versions of color recognition and facial recognitioncould be a special search systems used by the blind.

The front end server system receives results from the parallel searchsystems (214). In some embodiments, the results are accompanied by asearch score. For some visual queries, some of the search systems willfind no relevant results. For example, if the visual query was a pictureof a flower, the facial recognition search system and the bar codesearch system will not find any relevant results. In some embodiments,if no relevant results are found, a null or zero search score isreceived from that search system (216). In some embodiments, if thefront end server does not receive a result from a search system after apre-defined period of time (e.g., 0.2, 0.5, 1, 2 or 5 seconds), it willprocess the received results as if that timed out server produced a nullsearch score and will process the received results from the other searchsystems.

Optionally, when at least two of the received search results meetpre-defined criteria, they are ranked (218). In some embodiments, one ofthe predefined criteria excludes void results. A pre-defined criterionis that the results are not void. In some embodiments, one of thepredefined criteria excludes results having numerical score (e.g., for arelevance factor) that falls below a pre-defined minimum score.Optionally, the plurality of search results are filtered (220). In someembodiments, the results are only filtered if the total number ofresults exceeds a pre-defined threshold. In some embodiments, all theresults are ranked but the results falling below a pre-defined minimumscore are excluded. For some visual queries, the content of the resultsare filtered. For example, if some of the results contain privateinformation or personal protected information, these results arefiltered out.

Optionally, the visual query server system creates a compound searchresult (222). One embodiment of this is when more than one search systemresult is embedded in an interactive results document as explained withrespect to FIG. 3. The term query server system (118, FIG. 1) mayaugment the results from one of the parallel search systems with resultsfrom a term search, where the additional results are either links todocuments or information sources, or text and/or images containingadditional information that may be relevant to the visual query. Thus,for example, the compound search result may contain an OCR result and alink to a named entity in the OCR document (224).

In some embodiments, the OCR search system (112-B, FIG. 1) or the frontend visual query processing server (110, FIG. 1) recognizes likelyrelevant words in the text. For example, it may recognize named entitiessuch as famous people or places. The named entities are submitted asquery terms to the term query server system (118, FIG. 1). In someembodiments, the term query results produced by the term query serversystem are embedded in the visual query result as a “link.” In someembodiments, the term query results are returned as separate links. Forexample, if a picture of a book cover were the visual query, it islikely that an object recognition search system will produce a highscoring hit for the book. As such a term query for the title of the bookwill be run on the term query server system 118 and the term queryresults are returned along with the visual query results. In someembodiments, the term query results are presented in a labeled group todistinguish them from the visual query results. The results may besearched individually, or a search may be performed using all therecognized named entities in the search query to produce particularlyrelevant additional search results. For example, if the visual query isa scanned travel brochure about Paris, the returned result may includelinks to the term query server system 118 for initiating a search on aterm query “Notre Dame.” Similarly, compound search results includeresults from text searches for recognized famous images. For example, inthe same travel brochure, live links to the term query results forfamous destinations shown as pictures in the brochure like “EiffelTower” and “Louvre” may also be shown (even if the terms “Eiffel Tower”and “Louvre” did not appear in the brochure itself.)

The visual query server system then sends at least one result to theclient system (226). Typically, if the visual query processing serverreceives a plurality of search results from at least some of theplurality of search systems, it will then send at least one of theplurality of search results to the client system. For some visualqueries, only one search system will return relevant results. Forexample, in a visual query containing only an image of text, only theOCR server's results may be relevant. For some visual queries, only oneresult from one search system may be relevant. For example, only theproduct related to a scanned bar code may be relevant. In theseinstances, the front end visual processing server will return only therelevant search result(s). For some visual queries, a plurality ofsearch results are sent to the client system, and the plurality ofsearch results include search results from more than one of the parallelsearch systems (228). This may occur when more than one distinct imageis in the visual query. For example, if the visual query were a pictureof a person riding a horse, results for facial recognition of the personcould be displayed along with object identification results for thehorse. In some embodiments, all the results for a particular query byimage search system are grouped and presented together. For example, thetop N facial recognition results are displayed under a heading “facialrecognition results” and the top N object recognition results aredisplayed together under a heading “object recognition results.”Alternatively, as discussed below, the search results from a particularimage search system may be grouped by image region. For example, if thevisual query includes two faces, both of which produce facialrecognition results, the results for each face would be presented as adistinct group. For some visual queries (e.g., a visual query includingan image of both text and one or more objects), the search results mayinclude both OCR results and one or more image-match results (230).

In some embodiments, the user may wish to learn more about a particularsearch result. For example, if the visual query was a picture of adolphin and the “image to terms” search system returns the followingterms “water,” “dolphin,” “blue,” and “Flipper;” the user may wish torun a text based query term search on “Flipper.” When the user wishes torun a search on a term query (e.g., as indicated by the user clicking onor otherwise selecting a corresponding link in the search results), thequery term server system (118, FIG. 1) is accessed, and the search onthe selected term(s) is run. The corresponding search term results aredisplayed on the client system either separately or in conjunction withthe visual query results (232). In some embodiments, the front endvisual query processing server (110, FIG. 1) automatically (i.e.,without receiving any user command, other than the initial visual query)chooses one or more top potential text results for the visual query,runs those text results on the term query server system 118, and thenreturns those term query results along with the visual query result tothe client system as a part of sending at least one search result to theclient system (232). In the example above, if “Flipper” was the firstterm result for the visual query picture of a dolphin, the front endserver runs a term query on “Flipper” and returns those term queryresults along with the visual query results to the client system. Thisembodiment, wherein a term result that is considered likely to beselected by the user is automatically executed prior to sending searchresults from the visual query to the user, saves the user time. In someembodiments, these results are displayed as a compound search result(222) as explained above. In other embodiments, the results are part ofa search result list instead of or in addition to a compound searchresult.

FIG. 3 is a flow diagram illustrating the process for responding to avisual query with an interactive results document. The first threeoperations (202, 210, 214) are described above with reference to FIG. 2.From the search results which are received from the parallel searchsystems (214), an interactive results document is created (302).

Creating the interactive results document (302) will now be described indetail. For some visual queries, the interactive results documentincludes one or more visual identifiers of respective sub-portions ofthe visual query. Each visual identifier has at least one userselectable link to at least one of the search results. A visualidentifier identifies a respective sub-portion of the visual query. Forsome visual queries, the interactive results document has only onevisual identifier with one user selectable link to one or more results.In some embodiments, a respective user selectable link to one or more ofthe search results has an activation region, and the activation regioncorresponds to the sub-portion of the visual query that is associatedwith a corresponding visual identifier.

In some embodiments, the visual identifier is a bounding box (304). Insome embodiments, the bounding box encloses a sub-portion of the visualquery as shown in FIG. 12A. The bounding box need not be a square orrectangular box shape but can be any sort of shape including circular,oval, conformal (e.g., to an object in, entity in or region of thevisual query), irregular or any other shape as shown in FIG. 12B. Forsome visual queries, the bounding box outlines the boundary of anidentifiable entity in a sub-portion of the visual query (306). In someembodiments, each bounding box includes a user selectable link to one ormore search results, where the user selectable link has an activationregion corresponding to a sub-portion of the visual query surrounded bythe bounding box. When the space inside the bounding box (the activationregion of the user selectable link) is selected by the user, searchresults that correspond to the image in the outlined sub-portion arereturned.

In some embodiments, the visual identifier is a label (307) as shown inFIG. 14. In some embodiments, label includes at least one termassociated with the image in the respective sub-portion of the visualquery. Each label is formatted for presentation in the interactiveresults document on or near the respective sub-portion. In someembodiments, the labels are color coded.

In some embodiments, each respective visual identifiers is formatted forpresentation in a visually distinctive manner in accordance with a typeof recognized entity in the respective sub-portion of the visual query.For example, as shown in FIG. 13, bounding boxes around a product, aperson, a trademark, and the two textual areas are each presented withdistinct cross-hatching patterns, representing differently coloredtransparent bounding boxes. In some embodiments, the visual identifiersare formatted for presentation in visually distinctive manners such asoverlay color, overlay pattern, label background color, label backgroundpattern, label font color, and border color.

In some embodiments, the user selectable link in the interactive resultsdocument is a link to a document or object that contains one or moreresults related to the corresponding sub-portion of the visual query(308). In some embodiments, at least one search result includes datarelated to the corresponding sub-portion of the visual query. As such,when the user selects the selectable link associated with the respectivesub-portion, the user is directed to the search results corresponding tothe recognized entity in the respective sub-portion of the visual query.

For example, if a visual query was a photograph of a bar code, there maybe portions of the photograph which are irrelevant parts of thepackaging upon which the bar code was affixed. The interactive resultsdocument may include a bounding box around only the bar code. When theuser selects inside the outlined bar code bounding box, the bar codesearch result is displayed. The bar code search result may include oneresult, the name of the product corresponding to that bar code, or thebar code results may include several results such as a variety of placesin which that product can be purchased, reviewed, etc.

In some embodiments, when the sub-portion of the visual querycorresponding to a respective visual identifier contains text comprisingone or more terms, the search results corresponding to the respectivevisual identifier include results from a term query search on at leastone of the terms in the text. In some embodiments, when the sub-portionof the visual query corresponding to a respective visual identifiercontains a person's face for which at least one match (i.e., searchresult) is found that meets predefined reliability (or other) criteria,the search results corresponding to the respective visual identifierinclude one or more of: name, handle, contact information, accountinformation, address information, current location of a related mobiledevice associated with the person whose face is contained in theselectable sub-portion, other images of the person whose face iscontained in the selectable sub-portion, and potential image matches forthe person's face. In some embodiments, when the sub-portion of thevisual query corresponding to a respective visual identifier contains aproduct for which at least one match (i.e., search result) is found thatmeets predefined reliability (or other) criteria, the search resultscorresponding to the respective visual identifier include one or moreof: product information, a product review, an option to initiatepurchase of the product, an option to initiate a bid on the product, alist of similar products, and a list of related products.

Optionally, a respective user selectable link in the interactive resultsdocument includes anchor text, which is displayed in the documentwithout having to activate the link. The anchor text providesinformation, such as a key word or term, related to the informationobtained when the link is activated. Anchor text may be displayed aspart of the label (307), or in a portion of a bounding box (304), or asadditional information displayed when a user hovers a cursor over a userselectable link for a pre-determined period of time such as 1 second.

Optionally, a respective user selectable link in the interactive resultsdocument is a link to a search engine for searching for information ordocuments corresponding to a text-based query (sometimes herein called aterm query). Activation of the link causes execution of the search bythe search engine, where the query and the search engine are specifiedby the link (e.g., the search engine is specified by a URL in the linkand the text-based search query is specified by a URL parameter of thelink), with results returned to the client system. Optionally, the linkin this example may include anchor text specifying the text or terms inthe search query.

In some embodiments, the interactive results document produced inresponse to a visual query can include a plurality of links thatcorrespond to results from the same search system. For example, a visualquery may be an image or picture of a group of people. The interactiveresults document may include bounding boxes around each person, whichwhen activated returns results from the facial recognition search systemfor each face in the group. For some visual queries, a plurality oflinks in the interactive results document corresponds to search resultsfrom more than one search system (310). For example, if a picture of aperson and a dog was submitted as the visual query, bounding boxes inthe interactive results document may outline the person and the dogseparately. When the person (in the interactive results document) isselected, search results from the facial recognition search system areretuned, and when the dog (in the interactive results document) isselected, results from the image-to-terms search system are returned.For some visual queries, the interactive results document contains anOCR result and an image match result (312). For example, if a picture ofa person standing next to a sign were submitted as a visual query, theinteractive results document may include visual identifiers for theperson and for the text in the sign. Similarly, if a scan of a magazinewas used as the visual query, the interactive results document mayinclude visual identifiers for photographs or trademarks inadvertisements on the page as well as a visual identifier for the textof an article also on that page.

After the interactive results document has been created, it is sent tothe client system (314). In some embodiments, the interactive resultsdocument (e.g., document 1200, FIG. 15) is sent in conjunction with alist of search results from one or more parallel search systems, asdiscussed above with reference to FIG. 2. In some embodiments, theinteractive results document is displayed at the client system above orotherwise adjacent to a list of search results from one or more parallelsearch systems (315) as shown in FIG. 15.

Optionally, the user will interact with the results document byselecting a visual identifier in the results document. The server systemreceives from the client system information regarding the user selectionof a visual identifier in the interactive results document (316). Asdiscussed above, in some embodiments, the link is activated by selectingan activation region inside a bounding box. In other embodiments, thelink is activated by a user selection of a visual identifier of asub-portion of the visual query, which is not a bounding box. In someembodiments, the linked visual identifier is a hot button, a labellocated near the sub-portion, an underlined word in text, or otherrepresentation of an object or subject in the visual query.

In embodiments where the search results list is presented with theinteractive results document (315), when the user selects a userselectable link (316), the search result in the search results listcorresponding to the selected link is identified. In some embodiments,the cursor will jump or automatically move to the first resultcorresponding to the selected link. In some embodiments in which thedisplay of the client 102 is too small to display both the interactiveresults document and the entire search results list, selecting a link inthe interactive results document causes the search results list toscroll or jump so as to display at least a first result corresponding tothe selected link. In some other embodiments, in response to userselection of a link in the interactive results document, the resultslist is reordered such that the first result corresponding to the linkis displayed at the top of the results list.

In some embodiments, when the user selects the user selectable link(316) the visual query server system sends at least a subset of theresults, related to a corresponding sub-portion of the visual query, tothe client for display to the user (318). In some embodiments, the usercan select multiple visual identifiers concurrently and will receive asubset of results for all of the selected visual identifiers at the sametime. In other embodiments, search results corresponding to the userselectable links are preloaded onto the client prior to user selectionof any of the user selectable links so as to provide search results tothe user virtually instantaneously in response to user selection of oneor more links in the interactive results document.

FIG. 4 is a flow diagram illustrating the communications between aclient and a visual query server system. The client 102 receives avisual query from a user/querier (402). In some embodiments, visualqueries can only be accepted from users who have signed up for or “optedin” to the visual query system. In some embodiments, searches for facialrecognition matches are only performed for users who have signed up forthe facial recognition visual query system, while other types of visualqueries are performed for anyone regardless of whether they have “optedin” to the facial recognition portion.

As explained above, the format of the visual query can take many forms.The visual query will likely contain one or more subjects located insub-portions of the visual query document. For some visual queries, theclient system 102 performs type recognition pre-processing on the visualquery (404). In some embodiments, the client system 102 searches forparticular recognizable patterns in this pre-processing system. Forexample, for some visual queries the client may recognize colors. Forsome visual queries the client may recognize that a particularsub-portion is likely to contain text (because that area is made up ofsmall dark characters surrounded by light space etc.) The client maycontain any number of pre-processing type recognizers, or typerecognition modules. In some embodiments, the client will have a typerecognition module (barcode recognition 406) for recognizing bar codes.It may do so by recognizing the distinctive striped pattern in arectangular area. In some embodiments, the client will have a typerecognition module (face detection 408) for recognizing that aparticular subject or sub-portion of the visual query is likely tocontain a face.

In some embodiments, the recognized “type” is returned to the user forverification. For example, the client system 102 may return a messagestating “a bar code has been found in your visual query, are youinterested in receiving bar code query results?” In some embodiments,the message may even indicate the sub-portion of the visual query wherethe type has been found. In some embodiments, this presentation issimilar to the interactive results document discussed with reference toFIG. 3. For example, it may outline a sub-portion of the visual queryand indicate that the sub-portion is likely to contain a face, and askthe user if they are interested in receiving facial recognition results.

After the client 102 performs the optional pre-processing of the visualquery, the client sends the visual query to the visual query serversystem 106, specifically to the front end visual query processing server110. In some embodiments, if pre-processing produced relevant results,i.e., if one of the type recognition modules produced results above acertain threshold, indicating that the query or a sub-portion of thequery is likely to be of a particular type (face, text, barcode etc.),the client will pass along information regarding the results of thepre-processing. For example, the client may indicate that the facerecognition module is 75% sure that a particular sub-portion of thevisual query contains a face. More generally, the pre-processingresults, if any, include one or more subject type values (e.g., barcode, face, text, etc.). Optionally, the pre-processing results sent tothe visual query server system include one or more of: for each subjecttype value in the pre-processing results, information identifying asub-portion of the visual query corresponding to the subject type value,and for each subject type value in the pre-processing results, aconfidence value indicating a level of confidence in the subject typevalue and/or the identification of a corresponding sub-portion of thevisual query.

The front end server 110 receives the visual query from the clientsystem (202). The visual query received may contain the pre-processinginformation discussed above. As described above, the front end serversends the visual query to a plurality of parallel search systems (210).If the front end server 110 received pre-processing informationregarding the likelihood that a sub-portion contained a subject of acertain type, the front end server may pass this information along toone or more of the parallel search systems. For example, it may pass onthe information that a particular sub-portion is likely to be a face sothat the facial recognition search system 112-A can process thatsubsection of the visual query first. Similarly, sending the sameinformation (that a particular sub-portion is likely to be a face) maybe used by the other parallel search systems to ignore that sub-portionor analyze other sub-portions first. In some embodiments, the front endserver will not pass on the pre-processing information to the parallelsearch systems, but will instead use this information to augment the wayin which it processes the results received from the parallel searchsystems.

As explained with reference to FIG. 2, for at some visual queries, thefront end server 110 receives a plurality of search results from theparallel search systems (214). The front end server may then perform avariety of ranking and filtering, and may create an interactive searchresult document as explained with reference to FIGS. 2 and 3. If thefront end server 110 received pre-processing information regarding thelikelihood that a sub-portion contained a subject of a certain type, itmay filter and order by giving preference to those results that matchthe pre-processed recognized subject type. If the user indicated that aparticular type of result was requested, the front end server will takethe user's requests into account when processing the results. Forexample, the front end server may filter out all other results if theuser only requested bar code information, or the front end server willlist all results pertaining to the requested type prior to listing theother results. If an interactive visual query document is returned, theserver may pre-search the links associated with the type of result theuser indicated interest in, while only providing links for performingrelated searches for the other subjects indicated in the interactiveresults document. Then the front end server 110 sends the search resultsto the client system (226).

The client 102 receives the results from the server system (412). Whenapplicable, these results will include the results that match the typeof result found in the pre-processing stage. For example, in someembodiments they will include one or more bar code results (414) or oneor more facial recognition results (416). If the client's pre-processingmodules had indicated that a particular type of result was likely, andthat result was found, the found results of that type will be listedprominently.

Optionally the user will select or annotate one or more of the results(418). The user may select one search result, may select a particulartype of search result, and/or may select a portion of an interactiveresults document (420). Selection of a result is implicit feedback thatthe returned result was relevant to the query. Such feedback informationcan be utilized in future query processing operations. An annotationprovides explicit feedback about the returned result that can also beutilized in future query processing operations. Annotations take theform of corrections of portions of the returned result (like acorrection to a mis-OCRed word) or a separate annotation (either freeform or structured.)

The user's selection of one search result, generally selecting the“correct” result from several of the same type (e.g., choosing thecorrect result from a facial recognition server), is a process that isreferred to as a selection among interpretations. The user's selectionof a particular type of search result, generally selecting the result“type” of interest from several different types of returned results(e.g., choosing the OCRed text of an article in a magazine rather thanthe visual results for the advertisements also on the same page), is aprocess that is referred to as disambiguation of intent. A user maysimilarly select particular linked words (such as recognized namedentities) in an OCRed document as explained in detail with reference toFIG. 8.

The user may alternatively or additionally wish to annotate particularsearch results. This annotation may be done in freeform style or in astructured format (422). The annotations may be descriptions of theresult or may be reviews of the result. For example, they may indicatethe name of subject(s) in the result, or they could indicate “this is agood book” or “this product broke within a year of purchase.” Anotherexample of an annotation is a user-drawn bounding box around asub-portion of the visual query and user-provided text identifying theobject or subject inside the bounding box. User annotations areexplained in more detail with reference to FIG. 5.

The user selections of search results and other annotations are sent tothe server system (424). The front end server 110 receives theselections and annotations and further processes them (426). If theinformation was a selection of an object, sub-region or term in aninteractive results document, further information regarding thatselection may be requested, as appropriate. For example, if theselection was of one visual result, more information about that visualresult would be requested. If the selection was a word (either from theOCR server or from the Image-to-Terms server) a textual search of thatword would be sent to the term query server system 118. If the selectionwas of a person from a facial image recognition search system, thatperson's profile would be requested. If the selection was for aparticular portion of an interactive search result document, theunderlying visual query results would be requested.

If the server system receives an annotation, the annotation is stored ina query and annotation database 116, explained with reference to FIG. 5.Then the information from the annotation database 116 is periodicallycopied to individual annotation databases for one or more of theparallel server systems, as discussed below with reference to FIGS.7-10.

FIG. 5 is a block diagram illustrating a client system 102 in accordancewith one embodiment of the present invention. The client system 102typically includes one or more processing units (CPU's) 702, one or morenetwork or other communications interfaces 704, memory 712, and one ormore communication buses 714 for interconnecting these components. Theclient system 102 includes a user interface 705. The user interface 705includes a display device 706 and optionally includes an input meanssuch as a keyboard, mouse, or other input buttons 708. Alternatively orin addition the display device 706 includes a touch sensitive surface709, in which case the display 706/709 is a touch sensitive display. Inclient systems that have a touch sensitive display 706/709, a physicalkeyboard is optional (e.g., a soft keyboard may be displayed whenkeyboard entry is needed). Furthermore, some client systems use amicrophone and voice recognition to supplement or replace the keyboard.Optionally, the client 102 includes a GPS (global positioning satellite)receiver, or other location detection apparatus 707 for determining thelocation of the client system 102. In some embodiments, visual querysearch services are provided that require the client system 102 toprovide the visual query server system to receive location informationindicating the location of the client system 102.

The client system 102 also includes an image capture device 710 such asa camera or scanner. Memory 712 includes high-speed random accessmemory, such as DRAM, SRAM, DDR RAM or other random access solid statememory devices; and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 712 may optionally include one or more storage devices remotelylocated from the CPU(s) 702. Memory 712, or alternately the non-volatilememory device(s) within memory 712, comprises a non-transitory computerreadable storage medium. In some embodiments, memory 712 or the computerreadable storage medium of memory 712 stores the following programs,modules and data structures, or a subset thereof:

-   -   an operating system 716 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 718 that is used for connecting        the client system 102 to other computers via the one or more        communication network interfaces 704 (wired or wireless) and one        or more communication networks, such as the Internet, other wide        area networks, local area networks, metropolitan area networks,        and so on;    -   a image capture module 720 for processing a respective image        captured by the image capture device/camera 710, where the        respective image may be sent (e.g., by a client application        module) as a visual query to the visual query server system;    -   one or more client application modules 722 for handling various        aspects of querying by image, including but not limited to: a        query-by-image submission module 724 for submitting visual        queries to the visual query server system; optionally a region        of interest selection module 725 that detects a selection (such        as a gesture on the touch sensitive display 706/709) of a region        of interest in an image and prepares that region of interest as        a visual query; a results browser 726 for displaying the results        of the visual query; and optionally an annotation module 728        with optional modules for structured annotation text entry 730        such as filling in a form or for freeform annotation text entry        732, which can accept annotations from a variety of formats, and        an image region selection module 734 (sometimes referred to        herein as a result selection module) which allows a user to        select a particular sub-portion of an image for annotation;    -   an optional content authoring application(s) 736 that allow a        user to author a visual query by creating or editing an image        rather than just capturing one via the image capture device 710;        optionally, one or such applications 736 may include        instructions that enable a user to select a sub-portion of an        image for use as a visual query;    -   an optional local image analysis module 738 that pre-processes        the visual query before sending it to the visual query server        system. The local image analysis may recognize particular types        of images, or sub-regions within an image. Examples of image        types that may be recognized by such modules 738 include one or        more of: facial type (facial image recognized within visual        query), bar code type (bar code recognized within visual query),        and text type (text recognized within visual query); and    -   additional optional client applications 740 such as an email        application, a phone application, a browser application, a        mapping application, instant messaging application, social        networking application etc. In some embodiments, the application        corresponding to an appropriate actionable search result can be        launched or accessed when the actionable search result is        selected.

Optionally, the image region selection module 734 which allows a user toselect a particular sub-portion of an image for annotation, also allowsthe user to choose a search result as a “correct” hit withoutnecessarily further annotating it. For example, the user may bepresented with a top N number of facial recognition matches and maychoose the correct person from that results list. For some searchqueries, more than one type of result will be presented, and the userwill choose a type of result. For example, the image query may include aperson standing next to a tree, but only the results regarding theperson is of interest to the user. Therefore, the image selection module734 allows the user to indicate which type of image is the “correct”type—i.e., the type he is interested in receiving. The user may alsowish to annotate the search result by adding personal comments ordescriptive words using either the annotation text entry module 730 (forfilling in a form) or freeform annotation text entry module 732.

In some embodiments, the optional local image analysis module 738 is aportion of the client application (108, FIG. 1). Furthermore, in someembodiments the optional local image analysis module 738 includes one ormore programs to perform local image analysis to pre-process orcategorize the visual query or a portion thereof. For example, theclient application 722 may recognize that the image contains a bar code,a face, or text, prior to submitting the visual query to a searchengine. In some embodiments, when the local image analysis module 738detects that the visual query contains a particular type of image, themodule asks the user if they are interested in a corresponding type ofsearch result. For example, the local image analysis module 738 maydetect a face based on its general characteristics (i.e., withoutdetermining which person's face) and provides immediate feedback to theuser prior to sending the query on to the visual query server system. Itmay return a result like, “A face has been detected, are you interestedin getting facial recognition matches for this face?” This may save timefor the visual query server system (106, FIG. 1). For some visualqueries, the front end visual query processing server (110, FIG. 1) onlysends the visual query to the search system 112 corresponding to thetype of image recognized by the local image analysis module 738. Inother embodiments, the visual query to the search system 112 may sendthe visual query to all of the search systems 112A-N, but will rankresults from the search system 112 corresponding to the type of imagerecognized by the local image analysis module 738. In some embodiments,the manner in which local image analysis impacts on operation of thevisual query server system depends on the configuration of the clientsystem, or configuration or processing parameters associated with eitherthe user or the client system. Furthermore, the actual content of anyparticular visual query and the results produced by the local imageanalysis may cause different visual queries to be handled differently ateither or both the client system and the visual query server system.

In some embodiments, bar code recognition is performed in two steps,with analysis of whether the visual query includes a bar code performedon the client system at the local image analysis module 738. Then thevisual query is passed to a bar code search system only if the clientdetermines the visual query is likely to include a bar code. In otherembodiments, the bar code search system processes every visual query.

Optionally, the client system 102 includes additional clientapplications 740.

FIG. 6 is a block diagram illustrating a front end visual queryprocessing server system 110 in accordance with one embodiment of thepresent invention. The front end server 110 typically includes one ormore processing units (CPU's) 802, one or more network or othercommunications interfaces 804, memory 812, and one or more communicationbuses 814 for interconnecting these components. Memory 812 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and may include non-volatilememory, such as one or more magnetic disk storage devices, optical diskstorage devices, flash memory devices, or other non-volatile solid statestorage devices. Memory 812 may optionally include one or more storagedevices remotely located from the CPU(s) 802. Memory 812, or alternatelythe non-volatile memory device(s) within memory 812, comprises anon-transitory computer readable storage medium. In some embodiments,memory 812 or the computer readable storage medium of memory 812 storesthe following programs, modules and data structures, or a subsetthereof:

-   -   an operating system 816 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 818 that is used for connecting        the front end server system 110 to other computers via the one        or more communication network interfaces 804 (wired or wireless)        and one or more communication networks, such as the Internet,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   a query manager 820 for handling the incoming visual queries        from the client system 102 and sending them to two or more        parallel search systems; as described elsewhere in this        document, in some special situations a visual query may be        directed to just one of the search systems, such as when the        visual query includes an client-generated instruction (e.g.,        “facial recognition search only”);    -   a results filtering module 822 for optionally filtering the        results from the one or more parallel search systems and sending        the top or “relevant” results to the client system 102 for        presentation;    -   a results ranking and formatting module 824 for optionally        ranking the results from the one or more parallel search systems        and for formatting the results for presentation;    -   a results document creation module 826, is used when        appropriate, to create an interactive search results document;        module 826 may include sub-modules, including but not limited to        a bounding box creation module 828 and a link creation module        830;    -   a label creation module 831 for creating labels that are visual        identifiers of respective sub-portions of a visual query;    -   an annotation module 832 for receiving annotations from a user        and sending them to an annotation database 116;    -   an actionable search results module 838 for generating, in        response to a visual query, one or more actionable search result        elements, each configured to launch a client-side action;        examples of actionable search result elements are buttons to        initiate a telephone call, to initiate email message, to map an        address, to make a restaurant reservation, and to provide an        option to purchase a product; and    -   a query and annotation database 116 which comprises the database        itself 834 and an index to the database 836.

The results ranking and formatting module 824 ranks the results returnedfrom the one or more parallel search systems (112-A-112-N, FIG. 1). Asalready noted above, for some visual queries, only the results from onesearch system may be relevant. In such an instance, only the relevantsearch results from that one search system are ranked. For some visualqueries, several types of search results may be relevant. In theseinstances, in some embodiments, the results ranking and formattingmodule 824 ranks all of the results from the search system having themost relevant result (e.g., the result with the highest relevance score)above the results for the less relevant search systems. In otherembodiments, the results ranking and formatting module 824 ranks a topresult from each relevant search system above the remaining results. Insome embodiments, the results ranking and formatting module 824 ranksthe results in accordance with a relevance score computed for each ofthe search results. For some visual queries, augmented textual queriesare performed in addition to the searching on parallel visual searchsystems. In some embodiments, when textual queries are also performed,their results are presented in a manner visually distinctive from thevisual search system results.

The results ranking and formatting module 824 also formats the results.In some embodiments, the results are presented in a list format. In someembodiments, the results are presented by means of an interactiveresults document. In some embodiments, both an interactive resultsdocument and a list of results are presented. In some embodiments, thetype of query dictates how the results are presented. For example, ifmore than one searchable subject is detected in the visual query, thenan interactive results document is produced, while if only onesearchable subject is detected the results will be displayed in listformat only.

The results document creation module 826 is used to create aninteractive search results document. The interactive search resultsdocument may have one or more detected and searched subjects. Thebounding box creation module 828 creates a bounding box around one ormore of the searched subjects. The bounding boxes may be rectangularboxes, or may outline the shape(s) of the subject(s). The link creationmodule 830 creates links to search results associated with theirrespective subject in the interactive search results document. In someembodiments, clicking within the bounding box area activates thecorresponding link inserted by the link creation module.

The query and annotation database 116 contains information that can beused to improve visual query results. In some embodiments, the user mayannotate the image after the visual query results have been presented.Furthermore, in some embodiments the user may annotate the image beforesending it to the visual query search system. Pre-annotation may helpthe visual query processing by focusing the results, or running textbased searches on the annotated words in parallel with the visual querysearches. In some embodiments, annotated versions of a picture can bemade public (e.g., when the user has given permission for publication,for example by designating the image and annotation(s) as not private),so as to be returned as a potential image match hit. For example, if auser takes a picture of a flower and annotates the image by givingdetailed genus and species information about that flower, the user maywant that image to be presented to anyone who performs a visual queryresearch looking for that flower. In some embodiments, the informationfrom the query and annotation database 116 is periodically pushed to theparallel search systems 112, which incorporate relevant portions of theinformation (if any) into their respective individual databases 114.

FIG. 7 is a block diagram illustrating one of the parallel searchsystems utilized to process a visual query. FIG. 7 illustrates a“generic” server system 112-N in accordance with one embodiment of thepresent invention. This server system is generic only in that itrepresents any one of the visual query search servers 112-N. The genericserver system 112-N typically includes one or more processing units(CPU's) 502, one or more network or other communications interfaces 504,memory 512, and one or more communication buses 514 for interconnectingthese components. Memory 512 includes high-speed random access memory,such as DRAM, SRAM, DDR RAM or other random access solid state memorydevices; and may include non-volatile memory, such as one or moremagnetic disk storage devices, optical disk storage devices, flashmemory devices, or other non-volatile solid state storage devices.Memory 512 may optionally include one or more storage devices remotelylocated from the CPU(s) 502. Memory 512, or alternately the non-volatilememory device(s) within memory 512, comprises a non-transitory computerreadable storage medium. In some embodiments, memory 512 or the computerreadable storage medium of memory 512 stores the following programs,modules and data structures, or a subset thereof:

-   -   an operating system 516 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 518 that is used for connecting        the generic server system 112-N to other computers via the one        or more communication network interfaces 504 (wired or wireless)        and one or more communication networks, such as the Internet,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   a search application 520 specific to the particular server        system, it may for example be a bar code search application, a        color recognition search application, a product recognition        search application, an object-or-object category search        application, or the like;    -   an optional index 522 if the particular search application        utilizes an index;    -   an optional image database 524 for storing the images relevant        to the particular search application, where the image data        stored, if any, depends on the search process type;    -   an optional results ranking module 526 (sometimes called a        relevance scoring module) for ranking the results from the        search application, the ranking module may assign a relevancy        score for each result from the search application, and if no        results reach a pre-defined minimum score, may return a null or        zero value score to the front end visual query processing server        indicating that the results from this server system are not        relevant; and    -   an annotation module 528 for receiving annotation information        from an annotation database (116, FIG. 1) determining if any of        the annotation information is relevant to the particular search        application and incorporating any determined relevant portions        of the annotation information into the respective annotation        database 530.

FIG. 8 is a block diagram illustrating an OCR search system 112-Butilized to process a visual query in accordance with one embodiment ofthe present invention. The OCR search system 112-B typically includesone or more processing units (CPU's) 602, one or more network or othercommunications interfaces 604, memory 612, and one or more communicationbuses 614 for interconnecting these components. Memory 612 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and may include non-volatilememory, such as one or more magnetic disk storage devices, optical diskstorage devices, flash memory devices, or other non-volatile solid statestorage devices. Memory 612 may optionally include one or more storagedevices remotely located from the CPU(s) 602. Memory 612, or alternatelythe non-volatile memory device(s) within memory 612, comprises anon-transitory computer readable storage medium. In some embodiments,memory 612 or the computer readable storage medium of memory 612 storesthe following programs, modules and data structures, or a subsetthereof:

-   -   an operating system 616 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 618 that is used for connecting        the OCR search system 112-B to other computers via the one or        more communication network interfaces 604 (wired or wireless)        and one or more communication networks, such as the Internet,        other wide area networks, local area networks, metropolitan area        networks, and so on;    -   an Optical Character Recognition (OCR) module 620 which tries to        recognize text in the visual query, and converts the images of        letters into characters;    -   an optional OCR database 114-B which is utilized by the OCR        module 620 to recognize particular fonts, text patterns, and        other characteristics unique to letter recognition;    -   an optional spell check module 622 which improves the conversion        of images of letters into characters by checking the converted        words against a dictionary and replacing potentially        mis-converted letters in words that otherwise match a dictionary        word;    -   an optional named entity recognition module 624 which searches        for named entities within the converted text, sends the        recognized named entities as terms in a term query to the term        query server system (118, FIG. 1), and provides the results from        the term query server system as links embedded in the OCRed text        associated with the recognized named entities;    -   an optional text match application 632 which improves the        conversion of images of letters into characters by checking        converted segments (such as converted sentences and paragraphs)        against a database of text segments and replacing potentially        mis-converted letters in OCRed text segments that otherwise        match a text match application text segment, in some embodiments        the text segment found by the text match application is provided        as a link to the user (for example, if the user scanned one page        of the New York Times, the text match application may provide a        link to the entire posted article on the New York Times        website);    -   a results ranking and formatting module 626 for formatting the        OCRed results for presentation and formatting optional links to        named entities, and also optionally ranking any related results        from the text match application; and    -   an optional annotation module 628 for receiving annotation        information from an annotation database (116, FIG. 1)        determining if any of the annotation information is relevant to        the OCR search system and incorporating any determined relevant        portions of the annotation information into the respective        annotation database 630.

FIG. 9 is a block diagram illustrating a facial recognition searchsystem 112-A utilized to process a visual query in accordance with oneembodiment of the present invention. The facial recognition searchsystem 112-A typically includes one or more processing units (CPU's)902, one or more network or other communications interfaces 904, memory912, and one or more communication buses 914 for interconnecting thesecomponents. Memory 912 includes high-speed random access memory, such asDRAM, SRAM, DDR RAM or other random access solid state memory devices;and may include non-volatile memory, such as one or more magnetic diskstorage devices, optical disk storage devices, flash memory devices, orother non-volatile solid state storage devices. Memory 912 mayoptionally include one or more storage devices remotely located from theCPU(s) 902. Memory 912, or alternately the non-volatile memory device(s)within memory 912, comprises a non-transitory computer readable storagemedium. In some embodiments, memory 912 or the computer readable storagemedium of memory 912 stores the following programs, modules and datastructures, or a subset thereof:

-   -   an operating system 916 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 918 that is used for connecting        the facial recognition search system 112-A to other computers        via the one or more communication network interfaces 904 (wired        or wireless) and one or more communication networks, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   a facial recognition search application 920 for searching for        facial images matching the face(s) presented in the visual query        in a facial image database 114-A and searches the social network        database 922 for information regarding each match found in the        facial image database 114-A.    -   a facial image database 114-A for storing one or more facial        images for a plurality of users; optionally, the facial image        database includes facial images for people other than users,        such as family members and others known by users and who have        been identified as being present in images included in the        facial image database 114-A; optionally, the facial image        database includes facial images obtained from external sources,        such as vendors of facial images that are legally in the public        domain;    -   optionally, a social network database 922 which contains        information regarding users of the social network such as name,        address, occupation, group memberships, social network        connections, current GPS location of mobile device, share        preferences, interests, age, hometown, personal statistics, work        information, etc. as discussed in more detail with reference to        FIG. 12A;    -   a results ranking and formatting module 924 for ranking (e.g.,        assigning a relevance and/or match quality score to) the        potential facial matches from the facial image database 114-A        and formatting the results for presentation; in some        embodiments, the ranking or scoring of results utilizes related        information retrieved from the aforementioned social network        database; in some embodiment, the search formatted results        include the potential image matches as well as a subset of        information from the social network database; and    -   an annotation module 926 for receiving annotation information        from an annotation database (116, FIG. 1) determining if any of        the annotation information is relevant to the facial recognition        search system and storing any determined relevant portions of        the annotation information into the respective annotation        database 928.

FIG. 10 is a block diagram illustrating an image-to-terms search system112-C utilized to process a visual query in accordance with oneembodiment of the present invention. In some embodiments, theimage-to-terms search system recognizes objects (instance recognition)in the visual query. In other embodiments, the image-to-terms searchsystem recognizes object categories (type recognition) in the visualquery. In some embodiments, the image to terms system recognizes bothobjects and object-categories. The image-to-terms search system returnspotential term matches for images in the visual query. Theimage-to-terms search system 112-C typically includes one or moreprocessing units (CPU's) 1002, one or more network or othercommunications interfaces 1004, memory 1012, and one or morecommunication buses 1014 for interconnecting these components. Memory1012 includes high-speed random access memory, such as DRAM, SRAM, DDRRAM or other random access solid state memory devices; and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 1012 may optionallyinclude one or more storage devices remotely located from the CPU(s)1002. Memory 1012, or alternately the non-volatile memory device(s)within memory 1012, comprises a non-transitory computer readable storagemedium. In some embodiments, memory 1012 or the computer readablestorage medium of memory 1012 stores the following programs, modules anddata structures, or a subset thereof:

-   -   an operating system 1016 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   a network communication module 1018 that is used for connecting        the image-to-terms search system 112-C to other computers via        the one or more communication network interfaces 1004 (wired or        wireless) and one or more communication networks, such as the        Internet, other wide area networks, local area networks,        metropolitan area networks, and so on;    -   a image-to-terms search application 1020 that searches for        images matching the subject or subjects in the visual query in        the image search database 114-C;    -   an image search database 114-C which can be searched by the        search application 1020 to find images similar to the subject(s)        of the visual query;    -   a terms-to-image inverse index 1022, which stores the textual        terms used by users when searching for images using a text based        query search engine 1006;    -   a results ranking and formatting module 1024 for ranking the        potential image matches and/or ranking terms associated with the        potential image matches identified in the terms-to-image inverse        index 1022; and    -   an annotation module 1026 for receiving annotation information        from an annotation database (116, FIG. 1) determining if any of        the annotation information is relevant to the image-to terms        search system 112-C and storing any determined relevant portions        of the annotation information into the respective annotation        database 1028.

FIGS. 5-10 are intended more as functional descriptions of the variousfeatures which may be present in a set of computer systems than as astructural schematic of the embodiments described herein. In practice,and as recognized by those of ordinary skill in the art, items shownseparately could be combined and some items could be separated. Forexample, some items shown separately in these figures could beimplemented on single servers and single items could be implemented byone or more servers. The actual number of systems used to implementvisual query processing and how features are allocated among them willvary from one implementation to another.

Each of the methods described herein may be governed by instructionsthat are stored in a non-transitory computer readable storage medium andthat are executed by one or more processors of one or more servers orclients. The above identified modules or programs (i.e., sets ofinstructions) need not be implemented as separate software programs,procedures or modules, and thus various subsets of these modules may becombined or otherwise re-arranged in various embodiments. Each of theoperations shown in FIGS. 5-10 may correspond to instructions stored ina computer memory or non-transitory computer readable storage medium.

FIG. 11 illustrates a client system 102 with a screen shot of anexemplary visual query 1102. The client system 102 shown in FIG. 11 is amobile device such as a cellular telephone, portable music player, orportable emailing device. The client system 102 includes a display 706and one or more input means 708 such the buttons shown in this figure.In some embodiments, the display 706 is a touch sensitive display 709.In embodiments having a touch sensitive display 709, soft buttonsdisplayed on the display 709 may optionally replace some or all of theelectromechanical buttons 708. Touch sensitive displays are also helpfulin interacting with the visual query results as explained in more detailbelow. The client system 102 also includes an image capture mechanismsuch as a camera 710.

FIG. 11 illustrates a visual query 1102 which is a photograph or videoframe of a package on a shelf of a store. In the embodiments describedhere, the visual query is a two dimensional image having a resolutioncorresponding to the size of the visual query in pixels in each of twodimensions. The visual query 1102 in this example is a two dimensionalimage of three dimensional objects. The visual query 1102 includesbackground elements, a product package 1104, and a variety of types ofentities on the package including an image of a person 1106, an image ofa trademark 1108, an image of a product 1110, and a variety of textualelements 1112.

As explained with reference to FIG. 3, the visual query 1102 is sent tothe front end server 110, which sends the visual query 1102 to aplurality of parallel search systems (112A-N), receives the results andcreates an interactive results document.

FIGS. 12A and 12B each illustrate a client system 102 with a screen shotof an embodiment of an interactive results document 1200. Theinteractive results document 1200 includes one or more visualidentifiers 1202 of respective sub-portions of the visual query 1102,which each include a user selectable link to a subset of search results.FIGS. 12A and 12B illustrate an interactive results document 1200 withvisual identifiers that are bounding boxes 1202 (e.g., bounding boxes1202-1, 1202-2, 1202-3). In the embodiments shown in FIGS. 12A and 12B,the user activates the display of the search results corresponding to aparticular sub-portion by tapping on the activation region inside thespace outlined by its bounding box 1202. For example, the user wouldactivate the search results corresponding to the image of the person, bytapping on a bounding box 1306 (FIG. 13) surrounding the image of theperson. In other embodiments, the selectable link is selected using amouse or keyboard rather than a touch sensitive display. In someembodiments, the first corresponding search result is displayed when auser previews a bounding box 1202 (i.e., when the user single clicks,taps once, or hovers a pointer over the bounding box). The useractivates the display of a plurality of corresponding search resultswhen the user selects the bounding box (i.e., when the user doubleclicks, taps twice, or uses another mechanism to indicate selection.)

In FIGS. 12A and 12B the visual identifiers are bounding boxes 1202surrounding sub-portions of the visual query. FIG. 12A illustratesbounding boxes 1202 that are square or rectangular. FIG. 12B illustratesa bounding box 1202 that outlines the boundary of an identifiable entityin the sub-portion of the visual query, such as the bounding box 1202-3for a drink bottle. In some embodiments, a respective bounding box 1202includes smaller bounding boxes 1202 within it. For example, in FIGS.12A and 12B, the bounding box identifying the package 1202-1 surroundsthe bounding box identifying the trademark 1202-2 and all of the otherbounding boxes 1202. In some embodiments that include text, also includeactive hot links 1204 for some of the textual terms. FIG. 12B shows anexample where “Active Drink” and “United States” are displayed as hotlinks 1204. The search results corresponding to these terms are theresults received from the term query server system 118, whereas theresults corresponding to the bounding boxes are results from the queryby image search systems.

FIG. 13 illustrates a client system 102 with a screen shot of aninteractive results document 1200 that is coded by type of recognizedentity in the visual query. The visual query of FIG. 11 contains animage of a person 1106, an image of a trademark 1108, an image of aproduct 1110, and a variety of textual elements 1112. As such theinteractive results document 1200 displayed in FIG. 13 includes boundingboxes 1202 around a person 1306, a trademark 1308, a product 1310, andthe two textual areas 1312. The bounding boxes of FIG. 13 are eachpresented with separate cross-hatching which represents differentlycolored transparent bounding boxes 1202. In some embodiments, the visualidentifiers of the bounding boxes (and/or labels or other visualidentifiers in the interactive results document 1200) are formatted forpresentation in visually distinctive manners such as overlay color,overlay pattern, label background color, label background pattern, labelfont color, and bounding box border color. The type coding forparticular recognized entities is shown with respect to bounding boxesin FIG. 13, but coding by type can also be applied to visual identifiersthat are labels.

FIG. 14 illustrates a client device 102 with a screen shot of aninteractive results document 1200 with labels 1402 being the visualidentifiers of respective sub-portions of the visual query 1102 of FIG.11. The label visual identifiers 1402 each include a user selectablelink to a subset of corresponding search results. In some embodiments,the selectable link is identified by descriptive text displayed withinthe area of the label 1402. Some embodiments include a plurality oflinks within one label 1402. For example, in FIG. 14, the label hoveringover the image of a woman drinking includes a link to facial recognitionresults for the woman and a link to image recognition results for thatparticular picture (e.g., images of other products or advertisementsusing the same picture.)

In FIG. 14, the labels 1402 are displayed as partially transparent areaswith text that are located over their respective sub-portions of theinteractive results document. In other embodiments, a respective labelis positioned near but not located over its respective sub-portion ofthe interactive results document. In some embodiments, the labels arecoded by type in the same manner as discussed with reference to FIG. 13.In some embodiments, the user activates the display of the searchresults corresponding to a particular sub-portion corresponding to alabel 1302 by tapping on the activation region inside the space outlinedby the edges or periphery of the label 1302. The same previewing andselection functions discussed above with reference to the bounding boxesof FIGS. 12A and 12B also apply to the visual identifiers that arelabels 1402.

FIG. 15 illustrates a screen shot of an interactive results document1200 and the original visual query 1102 displayed concurrently with aresults list 1500. In some embodiments, the interactive results document1200 is displayed by itself as shown in FIGS. 12-14. In otherembodiments, the interactive results document 1200 is displayedconcurrently with the original visual query as shown in FIG. 15. In someembodiments, the list of visual query results 1500 is concurrentlydisplayed along with the original visual query 1102 and/or theinteractive results document 1200. The type of client system and theamount of room on the display 706 may determine whether the list ofresults 1500 is displayed concurrently with the interactive resultsdocument 1200. In some embodiments, the client system 102 receives (inresponse to a visual query submitted to the visual query server system)both the list of results 1500 and the interactive results document 1200,but only displays the list of results 1500 when the user scrolls belowthe interactive results document 1200. In some of these embodiments, theclient system 102 displays the results corresponding to a user selectedvisual identifier 1202/1402 without needing to query the server againbecause the list of results 1500 is received by the client system 102 inresponse to the visual query and then stored locally at the clientsystem 102.

In some embodiments, the list of results 1500 is organized intocategories 1502. Each category contains at least one result 1503. Insome embodiments, the categories titles are highlighted to distinguishthem from the results 1503. The categories 1502 are ordered according totheir calculated category weight. In some embodiments, the categoryweight is a combination of the weights of the highest N results in thatcategory. As such, the category that has likely produced more relevantresults is displayed first. In embodiments where more than one category1502 is returned for the same recognized entity (such as the facialimage recognition match and the image match shown in FIG. 15) thecategory displayed first has a higher category weight.

As explained with respect to FIG. 3, in some embodiments, when aselectable link in the interactive results document 1200 is selected bya user of the client system 102, the cursor will automatically move tothe appropriate category 1502 or to the first result 1503 in thatcategory. Alternatively, when a selectable link in the interactiveresults document is selected by a user of the client system 102, thelist of results 1500 is re-ordered such that the category or categoriesrelevant to the selected link are displayed first. This is accomplished,for example, by either coding the selectable links with informationidentifying the corresponding search results, or by coding the searchresults to indicate the corresponding selectable links or to indicatethe corresponding result categories.

In some embodiments, the categories of the search results correspond tothe query-by-image search system that produce those search results. Forexample, in FIG. 15 some of the categories are product match 1506, logomatch 1508, facial recognition match 1510, image match 1512. Theoriginal visual query 1102 and/or an interactive results document 1200may be similarly displayed with a category title such as the query 1504.Similarly, results from any term search performed by the term queryserver may also be displayed as a separate category, such as web results1514. In other embodiments, more than one entity in a visual query willproduce results from the same query-by-image search system. For example,the visual query could include two different faces that would returnseparate results from the facial recognition search system. As such, insome embodiments, the categories 1502 are divided by recognized entityrather than by search system. In some embodiments, an image of therecognized entity is displayed in the recognized entity category header1502 such that the results for that recognized entity aredistinguishable from the results for another recognized entity, eventhough both results are produced by the same query by image searchsystem. For example, in FIG. 15, the product match category 1506includes two entity product entities and as such as two entitycategories 1502—a boxed product 1516 and a bottled product 1518, each ofwhich have a plurality of corresponding search results 1503. In someembodiments, the categories may be divided by recognized entities andtype of query-by-image system. For example, in FIG. 15, there are twoseparate entities that returned relevant results under the product matchcategory product.

In some embodiments, the results 1503 include thumbnail images. Forexample, as shown for the facial recognition match results in FIG. 15,small versions (also called thumbnail images) of the pictures of thefacial matches for “Actress X” and “Social Network Friend Y” aredisplayed along with some textual description such as the name of theperson in the image.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

1-24. (canceled)
 25. A computer-implemented method to perform searchesbased on digital images, comprising: receiving, by one or more computingdevices, an input to capture digital images; receiving, by one or morecomputing devices, a digital image from an image capturing module;determining, by the one or more computing devices, that a particulartype of image has been received, the particular type of image beingassociated with a particular search engine; providing, by one or morecomputing devices, a recommendation to select the particular searchengine to perform a search; receiving, by the one or more computingdevices, a selection of the particular search engine; in response toreceiving the selection, performing, by the one or more computingdevices, a data search with the selected search engine; and providing,by one or more computing devices, results of the data search.
 26. Thecomputer implemented method of claim 25, wherein the determination thata particular type of image has been received is based on pre-processinginformation.
 27. The computer implemented method of claim 26, whereinthe pre-processing information is based on an optical characterrecognition algorithm performed on the digital image.
 28. The computerimplemented method of claim 26, wherein the pre-processing informationis based on an image search.
 29. The computer implemented method ofclaim 25, wherein the particular search engine is based on one or moreof a shopping search, a text search, or an image search.
 30. Thecomputer implemented method of claim 25, wherein recommendationcomprises a plurality of search engines.
 31. The computer implementedmethod of claim 30, wherein the plurality of search engines comprises ashopping search, an image search, and a text search.
 32. A computerprogram product, comprising: a non-transitory computer-readable storagedevice having computer-executable program instructions embodied thereonthat when executed by one or more computing devices cause the computerto perform searches based on digital images, the computer-readableprogram instructions comprising: computer-executable instructions toreceive an input to capture digital images; computer-executableinstructions to receive a digital image from an image capturing module;computer-executable instructions to determine that a particular type ofimage has been received, the particular type of image being associatedwith a particular search engine; computer-executable instructions toprovide a recommendation to select the particular search engine toperform a search; computer-executable instructions to receive aselection of the particular search engine; computer-executableinstructions to perform a data search with the selected search engine inresponse to receiving the selection; and computer-executableinstructions to provide results of the data search.
 33. The computerprogram product of claim 32, wherein the determination that a particulartype of image has been received is based on pre-processing information.34. The computer program product of claim 33, wherein the pre-processinginformation is based on an optical character recognition algorithmperformed on the digital image.
 35. The computer program product ofclaim 33, wherein the pre-processing information is based on an imagesearch.
 36. The computer program product of claim 32, wherein theparticular search engine is based on one or more of a shopping search, atext search, or an image search.
 37. The computer program product ofclaim 32, wherein recommendation comprises a plurality of searchengines.
 38. The computer program product of claim 37, wherein theplurality of search engines comprises a shopping search, an imagesearch, and a text search.
 39. A system to perform searches based ondigital images, comprising: a storage device; and a processorcommunicatively coupled to the storage device, wherein the processorexecutes application code instructions that are stored in the storagedevice to cause the system to: receive an input to capture digitalimages; receive a digital image from an image capturing module;determine that a particular type of image has been received, theparticular type of image being associated with a particular searchengine; provide a recommendation to select the particular search engineto perform a search; receive a selection of the particular searchengine; perform a data search with the selected search engine inresponse to receiving the selection; and provide results of the datasearch.
 40. The system of claim 39, wherein the determination that aparticular type of image has been received is based on pre-processinginformation.
 41. The system of claim 40, wherein the pre-processinginformation is based on an optical character recognition algorithmperformed on the digital image.
 42. The system of claim 40, wherein thepre-processing information is based on an image search.
 43. The systemof claim 39, wherein the particular search engine is based on one ormore of a shopping search, a text search, or an image search.
 44. Thesystem of claim 39, wherein recommendation comprises a plurality ofsearch engines.